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Abstract 

We report findings on the methodological phase of a research project designed to 
assess the progress of the British Columbia Open University (BCOU) toward a 
1997 goal of increasing distance education course completion rates to British 
Columbia system levels by adapting existing 'off-line' courses for online delivery 
(a virtualization strategy). The method consists of identifying benchmarking 
factors statistically through the regression of 15 institutional factors on course 
completion rate data, and using these factors to establish 'off-line' course 
completion rate benchmarks. Off-line courses are print-based independent study 
courses providing instructional support through mail, e-mail, and telephone. The 
second phase of the project will benchmark BCOU online courses against their off- 
line counterparts. The 'off-line' completion rates data comprise 137 courses and 
23,709 course enrolments and represent a three-year period. Stepwise linear 
regression includes eight factors and accounts for a significant amount of total 
variation (46.8%). The regression organizes benchmarks effectively along three 
Course Level and Subject Matter classification factors. We set BCOU benchmarks 
accordingly and compare them to the BC system. We highlight some unexpected 
results, including off-line completion rate benchmarks increasing over time. This 
suggests that BCOU's off-line courses are making their own contribution toward 
BCOU's goal of increasing distance education course completion rates to British 
Columbia system levels. 

Resume 

Cet article rapporte les resultats de la phase methodologique d'un projet de 
recherche congu pour evaluer les progres de la British Columbia Open University 
(BCOU) vers son objectif d'augmenter les taux de completion des cours a distance 
en regard des niveaux du systeme universitaire de la Colombie Britannique, en 
adaptant les cours hors ligne pour en faire des cours en ligne (strategie de 
virtualisation). La methode consiste a identifier les facteurs d'etalonnage de fagon 
statistique grace a une regression sur quinze facteurs institutionnels a partir des 
donnees de completion des cours, et en utilisant ces facteurs pour etablir un 
etalonnage des taux de completion de cours hors ligne. Les cours hors ligne sont 
sur support imprime et de type auto-apprentissage avec encadrement par la 
poste, par courriel et par telephone. La deuxieme phase du projet etalonnera les 
cours en ligne de la BCOU en opposition a leurs contreparties hors ligne. Les 
donnees de taux de completion hors ligne proviennent de 137 cours et de 23 709 
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inscriptions sur une periode de trois ans. La regression lineaire comprend huit 
facteurs et rend compte d'un pourcentage significatif de la variation totale 
(46,8%). La regression organise I'etalonnage selon trois facteurs « Niveau de 
cours » et trois facteurs « Matieie ». Nous avons etabli I'etalonnage en 
consequence et 1' avons compare a celui du systeme universitaire de la Colombie 
Britannique. Nous mettons en exergue des resultats inattendus, comme 
I'augmentation dans le temps de I'etalonnage des taux de completion des cours 
hors ligne. Ceci suggere que les cours hors ligne de la BCOU apportent leur 
propre contribution au but d'atteindre les niveaux du systeme universitaire de la 
Colombie Britannique pour les taux de completion des cours. 


Introduction 

Historically, distance education courses have exhibited low course 
completion rates relative to 'traditional' face-to-face education (Beatty- 
Guenter 2001, Carr 2000, Holmberg 1995, Howell et ah, 2004, Keegan 
1986, Munro 1991, Paul 1986, 1990, Zajkowski 1997). Ten years ago the 
British Columbia Open University (BCOU) resolved to close the 
completion rates 'gap' by upgrading its distance education platform and 
adapting its off-line courses for online delivery. BCOU off-line courses are 
print-based independent study courses that provide instructional support 
through mail, e-mail, and telephone. Research in the late 1990s suggested 
high completion rates for on-line courses, including claims of 
'equivalency' with face-to-face course delivery (Card and Horton 1998, 
Eliesen and Black 1997, 1998a, 1998b, Fredda 2000a, 2000b, 2000c, 
McDonald and Gibson 1998, Rosberg 1997, Sener 1996). In 1997 the Open 
Learning Agency (OLA), the BCOU's parent institution, resolved to put 
together a comprehensive on-line distance education platform to increase 
course completion rates to provincial standards, with the proviso that it 
would do so without additional resources or without sacrificing access 
and flexibility. The OLA and BCOU stated goal was to increase 
completion rates "by an amount sufficient to ensure that within two years 
(i.e., by March 1999) they meet or exceed the minimum as established by 
the other institutions in the BC College, Institute and Agency system 
(controlling for program area and using system-wide key performance 
indicators.)" Open Learning Agency, p. 3, (1997). 

This paper reports on the first phase of a research project designed to 
assess BCOU's progress toward that stated goal. Here we develop a 
benchmarking method by regressing 15 institutional factors on successful 
course completion rates and we benchmark course completion rates for 
BCOU's off-line distance education courses accordingly. We compare 
these benchmarks to BC system statistics. The second phase of the project 
will benchmark BCOU's online distance education against their off-line 
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counterparts, and will examine the impact of online course delivery on 
course completion rates. Benchmarking is a necessary step to course 
completion rate comparisons since intra-institutional course completion 
rates are highly variable (Howell et al., 2004.) 

The British Columbia Open University 

In Canada, education is a provincial or territorial jurisdiction. At the 
beginning of the millennium, the Province of British Columbia's Minisfry 
of Advanced Education funded and supporfed 25 posfsecondary 
institutions of which the BCOU was a constituent member. At that time, 
BCOU's course enrolment ranged between 20-25,000 and it served 12- 
14,000 students annually before if ceased to operate on March 31, 2005; its 
operations were then taken over by Thompson Rivers University (TRU). 

In its final days, fhe BCOU was a comparable insfifution fo 
TeleUniversife (Quebec) and Afhabasca Universify (Alberfa). If was a 
single mode distance education institution offering flexible learning 
opporfunifies fhrough credif banking and disfance education. BCOU's 
course enrolment was slightly less than TeleUniversite and significantly 
less than Athabasca University (by a factor of abouf fwo and a half). 

The British Coiumbia Open University Off-iine Independent Study Courses 
BCOU off-line independent study courses rely extensively on print, 
audio, and video instructional materials as well as textbooks. Students are 
given the choice of confacfing a fufor via mail, felephone, or e-mail. 
Telephone calls are received primarily during scheduled fwice-weekly 
fufor office hours. Tufors are required fo respond fo sfudenf e-mails 
wifhin fwo working days as well as to return assignments submitted via 
e-mail within three working days. Though some students still participate 
in BCOU courses in the traditional fashion (i.e., wifhouf e-mail supporf), 
e-mail communication has become more and more prevalenf in sfudent- 
fufor and sfudenf-sfaff inferacfions. Administrafive support is available 
by phone during business hours (usually from 8:30 AM till 4:30 PM from 
Monday fo Friday). 


Methods 

Course completion rafes were calculafed for unique course, course 
delivery mefhod, and fufor combinations. Regression analysis was used 
fo idenfify which ones of 15 insfifufional facfors (independenf variables) 
were mosf strongly associated with course completion (dependent 
variable). Course completion rate benchmarks were then established 
according to regression factors that were found fo be sfafisfically 
significant, rather than choosing them subjectively. 
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Data 

The target population was students in academic areas (Arts, Science, and 
Business) who completed an off-line independenf study distance 
education course in the fiscal years 2002/03, 2003/04, and 2004/ 05 (this 
excludes laboratory courses.) During that period, the BCOU offered 70 
online courses and 137 off-line courses, a balance fhat will enable us fo 
compare online courses to off-line courses in fhe second phase of this 
project. The analysis comprises 548 data based on 23,709 course 
registrations. A datum represents a unique course and tutor combination 
comprising 6 or more enrolments. 

Defining Compietion Rates 

Though 'traditional' institutions and 'open and distance education' 
institutions use distinctive grading scales and course completion statistics 
especially with regard to the definition of incomplete grades and 
withdrawal grades, the two systems can be linked through the concept of 
stable enrolment, viz., students who demonstrably started a course. 
Whereas most BC institutions identify 'starfers' using a fime-dependent 
approach (a cuf-off date chosen so as to ensure that statistics are based on 
students who clearly made an attempt to start a course). Open 
Universities such as the BCOU identify sfarfers wifh respecf fo outcome, 
viz., the fact that a student has completed some graded academic work, 
regardless of date. Through the concept of stable enrolment, the 
successful completion rate of "sfarters" links the grading systems of 'open 
and disfance education' institutions and of 'fradifional' insfitutions and 
fheir comparative performance (Woodley and Parlett 1983, Paul 1990, 
Howell ef. al, 2004). 

BCOU's successful complefion rafes (SCR) were fherefore calculated as 
follows: 

The total number of students who obtained a passing grade on an ABCD 
scale, divided by the total number of students who started (i.e., the number 
of students who obtained a passing grade on the ABCD scale plus those 
who received an Incomplete or a Fail grade) times 100. 

This course completion rate statistic excludes "W" grades as defined 
by 'open and distance education' institutions like the BCOU, i.e., a grade 
given to students who did not complete any graded academic work. 

Analysis 

Stepwise linear regression was used to analyze the data and only those 
factors associated with SCRs at a 0.05 level of sfafisfical significance are 
reporfed here. 
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The Institutional Factors 

The institutional factors consist of course factors and tutor factors. The 
course factors are: Level, Subject Matter, Number of Credifs, Year of 
Delivery as well as a unique number for each course (Course Number). 
They are coded as follows: 


Course Factors 

Regression Coding 

Level (3 subfactors) 

Subfactor 1 : Non-credit (ABE*) = 0 and Credit = 1 Subfactor 2: 
ABE = 0, and 1 to 4 for university from 100-level to 400-level 
(year 1 to year 4) Subfactor 3: Introductory course = 1 Other 
courses = 0 (Introductory means first of a pair of first or second 
year courses, such as Introduction to Physics 1 and Introduction 
to Physics 2) 

Subject Matter 

1 to 5, see below 

No. of Credits 

Typically 3 credits (actuals) 

Year of Delivery 

By fiscal year, 1 = 2002/03, 2 = 2003/04, 3 = 2004/05 

Course Number 

1 to 177 (assigned according to alphanumeric order) 


* ABE = Adult Basic Education 


The Subject Matter is coded as follows: Applied Academic areas 
(Business Administration and Business computer courses) are coded as 1. 
Liberal Arts subject matters are coded from 2 to 4 according to perceived 
level of quantitative difficulty. 


Subject Matter 

Course Code 

Regression Coding 

Business 

ADMN, BUSM, WKPL 

1 

Humanities 

ENGL, FREN, HIST, HUMN, SPAN 

2 

Social Sciences 

ECON, GEOG, GEOL, POLI, PSYC, SOCI 

3 

Science and Mathematics 

BISC, ENST, ENVS, CHEM, MATH, PHYS 

4 


The tutor factors are: Course Load (the number of courses supported 
by a tutor for that year). Student Load (the number of students supported 
by a tutor for that year). Tutor Age (in 2002), Years of Service at OLA, 
Gender, Place of Residence, Academic Qualifications, as well as a unique 
number for each tutor (Tutor Number). They are coded as follows: 
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Tutor Factors 

Regression Coding 

Course Load for 
Tutor in that Year 

1 to 9 courses (actuals) 

Student Load for 
Tutor in that Year 

5 to 688 students (actuals) 

Age in 2002 

In days (actuals) 

Years of Service in 
that Year 

In days since date of hire (actuals) 

Gender 

Female = 0 Male = 1 

Piace of Residence in 
2002/03 

1 = Metropolitan: Greater Vancouver and Greater Victoria 

2 = Urban: Kelowna, Kamloops, Nanaimo, Prince George 

3 = Rural: all other locations 

Academic Qualifications 

1 = PhD 2 = Master's 3 = Bachelor's or below 

Tutor Number 

1 to 130 (assigned according to alphabetical order) 


Results 

Identifying Benchmarking Factors from Among Institutional Factors 
The regression is summarized in Table 1. Eight steps account for 46.8% of 
fhe variation. This is a relatively high percentage considering that the data 
range widely (minimum 6.7% and maximum 100%). Previous attempts to 
regress factors on SCRs have accounted for 16.2% (Ostman and Wagner, 
1987) and 32% of the variation (Sweet, 1986; for Open Learning Insfifufe, 
fhe precursor institution to the OLA and the BCOU). 
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Table 1 : Linear Regression of Institutional Factors on SCRs 


Regression 

Step 

Factors 

F to Enter 

R2 

% of variation 
accounted for by 
regression 

1 

Course Level: Subfactor 2 
ABE to 400-level 

235.2 

0.302 

2 

Subject Matter 

156.0 

0.365 

3 

Course Level: Subfactor 1 
Credit vs Non-credit 

123.6 

0.406 

4 

Tutor Course Load 

102.7 

0.432 

5 

Course Levei: Subfactor 3 
introductory or Not 

86.2 

0.444 

6 

Course Number 

74.7 

0.454 

7 

Year of Deiivery 

65.9 

0.461 

8 

Tutor Gender 

59.1 

0.468 


Since Course Level and Subject Matter factors account for 40.6% of fhe 
variafion before fufor specific or course specific facfors are loaded, fhey 
represenf good prospecfs for benchmarking. 

Sfep 3 of the linear regression can be used to predict SCR benchmarks 
according to the general formula for regression: 

Y = a + (b • XI) + (c • X2) + (d • X3) Equation 1 


where XI represenfs Course Level: Subfacfor 2 codes, X2 fhe Subjecf 
Maffer codes, and X3 Course Level: Subfactor 1 codes. The regression 
coefficients are shown in Table 2. 
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Table 2: Linear Model for Predicting BCOU SCR Benchmarks Based on Course Level 
and Subject Matter factors (from Step 3 of regression) 


Factors 

R2 

Coefficient/Slope 

Intercept 


a = 61 .93 

Level (ABE to 400-level) 

0.302 

b = 4.81 

Subject Matter 

0.365 

c = -3.73 

Level (Credit or Non-credit) 

0.406 

d = 11.73 


This model generates the following equation and the following SCR 
benchmarks (shown in Table 3 against actuals): 

SCR = 61.93 + (4.81 • XI) + (-3.73 • X2) + (11.73 • X3) Equation 2 


Table 3: BCOU SCR Benchmarks*: Benchmarks over Actuals (sample size in 
parenthesis). (Benchmarks based on linear model described in Equation 2.) 


Successful Completion Rates (%), by Level 


Subject 

Matter 

Non-Credit 

Courses 


Credit Courses 



ABE 

100-Level 

200-Level 

300-Level 

400-Level 

Business 


74.7 

70.6 (51) 

79.6 

74.5 (16) 

84.4 

83.6 (28) 

89.2 

87.2(18) 

Humanities 

54.5 

52.8 (30) 

71.0 

74.3 (87) 

75.8 

80.2 (13) 

80.6 
83.7 (2) 

85.4 

85.7 (21) 

Social Sciences 

50.7 
55.2 (7) 

67.3 

74.1 (37) 

72.1 

73.0 (46) 

76.9 

80.9 (26) 

81.7 
81.4 (9) 

Science and Math 

47.0 

46.9 (66) 

63.6 

59.5 (74) 

68.4 

59.2 (12) 

73.2 

74.4(5) 



Off-line independent study academic courses with mail, e-mail, and telephone support 







BENCHMARKING COURSE COMPLETION RATES 


81 


The fit between the linear regression model and actuals is within a few 
percentage points for each pair. Two instances exceed 5 percentage points: 
first year Social Sciences actuals are 6.8 percentage points higher than 
prediction, and second year Science and Mathematics courses actuals are 
9.2 percentage points below prediction. This suggests a possible course 
design concern for BCOU's second year Science and Mathematics courses. 

Note that actuals shown in Table 3 are based on the same data used to 
generate the linear model itself. Thus they do not represent an 
independent test of the linear model. The fact that the two sets of data 
show remarkable linear congruence across five course level and four 
subject matter categories does however strongly support a benchmarking 
framework based on Course Level and Subject Matter classification. Note 
also that we could have successfully tested the 2002/03 data against 
independent 2003/04 or 2004/05 data. 

Other Significant Regression Factors 

Besides the three factors deemed to be effective in establishing BCOU's 
SCR benchmarks, five additional factors account for a statistically 
significant amount of SCR variation. They are Tutor Course Load and 
Introductory Course Factors (steps 4 and 5; an additional 3.8% of the 
variation accounted for), and Course Number, Year of Delivery, and Tutor 
Gender. These factors make a contribution to the regression over and 
above the contribution already attributed to the Course Level and Subject 
Matter benchmarking factors. 


Discussion 


Benchmarking Method 

This study set out to identify course completion rate benchmarks for 
BCOU off-line academic independent study courses through the 
regression of 15 available institutional factors against three years of 
successful completion rate data (SCR) computed for each unique 
individual course and tutor combination (for each datum comprising six 
or more enrolments). The multiple linear regression method used in this 
study has produced two significant findings: 

(1) It accounted for a large portion of the course completion rate 
variation (46.8%), and 

(2) It identified three appropriate benchmarking factors for two reasons: 
(a) The first three factors loading in the regression account for 40.6% 

of the variation (Table 2) and are generic or institution-wide 
(Course Level and Subject Matter), whereas tutor specific and 
course specific factors enter in later steps of the regression; and 
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(b) Actuals and benchmarks organized congruently across five 
Course Level and four Subject Matter classification factors, with 
little or no interaction between them (t = 0.975, p > 0.38). 

These results come with two provisos. First, because institutional 
factors subsume student characteristics, institutional and student factors 
are confounded. For example, SCRs may be high in upper-level courses 
because upper-level sfudenfs are on average more mature than first year 
university students, more educationally 'seasoned', more likely to have 
had prior distance education experience, or are 'survivors' of an academic 
aftrifion process. However, since fhe BCOU sfudent populafion has been 
relatively stable over a period of nearly fifteen years (Black 1992, 1993, 
1995a, 1995b, 1995c, 1996, Black et. al. 1994, Giguere 2002, Open Learning 
Agency 1988, 1989, 1990, 1991, 1992, 1993, 1994, 1995), the results 
presented here are robust. Second, a subject matter classification based on 
perceived level of quanfifafive difficulfy explains a considerable amounf 
of variafion for BCOU academic courses. This may not be the case for a 
differenf mix of subject matter or applied and academic programming 
combinations. Nevertheless Table 3 benchmarks illustrate the 
effectiveness of fhe BCOU independenf sfudy course design across course 
level and disciplinary audiences and appear adequafe fo compare 
BCOU's performance fo fhe BC sysfem, across similar audiences. 

Unexpected Results: SCR Benchmarks Increase Over Time 
On average, SCR benchmarks increased by 1.8 to 1.9 percentage points 
per year from 2002/03 fo 2004/05. This suggesfs fhaf BCOU's off-line 
course are making their own contribution toward BCOU's 1997 goal of 
increasing disfance education course completion rates to BC system 
levels. Given that at that time BCOU course offerings were sfable and the 
demographic characteristics of sfudents comparable, whaf could have 
caused this increase? We believe that e-mail communication played a 
significant role in this regard. By providing e-mail support options for 
studenfs in off-line courses, fhe BCOU provided fhe opporfunity for 
increasing immediacy of feedback, a factor that is known to have a 
positive effect on completion rates (Rekkedal 1985, Chang 2004, Gibson 
1997, Holmberg 1995, Howell et. al. 2004, Kember 1990, Mehrotra et. al. 
2001, Roberts 1984, Sweet 1986, Zajkowski 1997). BCOU student surveys 
support this interpretation. Student satisfaction with tutor contact 
frequency (rated on a scale of 1 to 4) averaged 2.74 in 1999 (out of 1,203 
responses) and of 3.28 in 2004/05 (ouf of 3,887 responses). This suggests 
that as more and more students become computer and e-mail savvy 
and / or the institution develops more comprehensive virtual services 
(e.g., online library, registration, financial aid, efc.), BCOU's off-line 
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courses will keep growing a virtual identity and BCOU's SCRs should 
continue to improve. In retrospect, it appears that BCOU's virtualization 
strategy comprises two components: the gradual virtualization of its off- 
line print-based independent study courses, and the adaptation of ifs off- 
line courses for online delivery. 

Other Unexpected Results 

To our knowledge, a negative association between SCRs and introductory 
courses (coefficient = -5.67, t = -3.47, p < 0.05) and a positive association 
between SCRs and Tutor Course Load (coefficient = 1.47, t = 4.90, p < 0.05) 
or Female Gender (coefficient = 3.16, t = 2.62, p < 0.05) have not been 
reported before. Because fhese results were unexpected and are 
somewhat counterintuitive, they provide useful infelligence from an 
insfifufional perspective: 

A. With regard to introductory courses, we expected SCRs to be greater 
than or equal to other courses at the same level, because introductory 
courses are high enrolment courses that are maintained and updated 
frequently. One possible explanation for low SCRs in infroductory 
courses is fhaf they may have disproportionately high numbers of 
firsf-time sfudents (the implication being that a first time distance 
education experience may be less successful). Dafa show however 
thaf first-time BCOU students take the same proportion of 
introductory courses (30.4%, 4,613 students) as second-time BCOU 
students (30.7%, 1,595 students.) Failing an adequate understanding 
as to why introductory courses exhibit low completion rates, 'Best 
Practices' could be used to address low completion rates in 

BCOU / Thompson Rivers University-Open Learning introductory 
courses. Relevant best practices range from additional face-to-face 
confacf, assignmenf structure, mentoring, counseling, liaison, peer 
contact, residency, community building, financial reward, and 
inferactive on-line delivery (Chang 2004, Gibson 1997, Howell et. al. 
2004, Kember 1990, Mehrotra ef. al. 2001, Roberts 1984, Sweet 1986, 
Zajkowski 1997). 

B. With regard to Tutor Course Load, our 'naive' expectation was that 
Course Load and/or Student Load would have a negative effect on 
SCRs. The finding fhaf Tufor Course Load is positively associated 
with SCRs surprised us. The linear contribution is 1.44 SCR 
percentage point per course, i.e., an additional SCR contribution of 
up fo 13 percentage points for a tutor supporting 11 courses. Since 
Tutor Years of Experience did nof significanfly confribute fo the 
regression, the results suggest that tutors carrying a broad course 
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portfolio increase SCRs by a significant amount. The institution 
needs to weigh this positive contribution against the benefits of 
developing a more diverse tutor body (e.g., providing to students 
with a richer academic experience, increasing tutor availability to 
address leave substitution, etc.) 

C. With regard to Tutor Gender, the data indicate that SCRs are 3.16 
percentage points higher for female tutors than for their male 
counterparts. Since we had no prior expectation at the onset of this 
project and are not aware of papers addressing this issue, we plan to 
re-visit it in phase two of this project where both off-line and online 
course data is available. 

BCOU's Performance Relative to BC System 

The purpose of this research project is to assess BCOU's progress toward 
the stated goal of increasing course completion rates to BC system levels 
by adapting existing off-line courses for online delivery. The first phase 
was designed to establish course completion rate benchmarks for off-line 
courses. Results show that BCOU off-line courses are making a small 
contribution toward BCOU's 1997 goal of increasing completion rates 
since off-line course SCRs are increasing by close to 2% per year. 

BCOU's off-line courses SCRs compare as follows (Table 4) to BC 
system statistics published near the time the BCOU set its goal to increase 
course completion rates to BC system levels (British Columbia 2000a, 
2000b). 

BCOU's SCR benchmarks are 23% below 1998 BC system level for off- 
line ABE courses whereas they are 10% below 1998 BC system level for 
off-line credit courses. These results set the stage to assess BCOU's 
progress toward its goal of increasing course completion rates to British 
Columbia system levels through online course delivery. 


Table 4: SCRs for the BC College, Institute and Agency System and BCOU 



ABE 


Upper 

Level 

BC College, Institute and 
Agency System, 1998 

72 

79 

93 

BCOU Off-line Course 
Benchmarks, 2002/03 

49 

70 

83 

Difference: 

23 

9 

10 
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